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Abstract 

College and universities evaluate the teaching performance of faculty members in a variety of 
ways. Benefits to effective faculty evaluation include advancing the scholarship of teaching and 
learning, as well as improving the functionality and innovation of courses, curriculum, 
departments, and ultimately the broader community (Boyer, 1990; Glassick, Huber, & Maeroff, 
1997). While there is ample research related to the evaluation of faculty in traditional settings, 
there have been fewer studies examining online faculty members’ perceptions of evaluation 
processes. Further, due to the growth of online education, the existing evaluation scales, 
including those used in traditional settings, have been called into question (Berk, 2013; Hathorn 
& Hathorn, 2010; Rothman, Romeo, Brennan, & Mitchell, 2011). This qualitative study 
examines one university’s online full-time faculty and their perceptions of the tools and 
processes used to evaluate their teaching. Through a systematic qualitative content analysis of 
survey data, findings indicate that online faculty members have a desire to grow as instructors, 
focusing little on modality or task-oriented expectations as a means for growth. Participants 
expressed an interest in holistic, descriptive evaluation feedback by a range of stakeholders, 
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particularly those with content knowledge. Study findings have implications for administrators 
and other stakeholders related to online full-time faculty, including the processes and documents 
through which they are evaluated. 

Keywords: evaluation, online faculty, full-time faculty, faculty evaluation, online learning, e- 
leaming, computer mediated learning 

INTRODUCTION 

The number of students participating in online learning is striking: 6.7 million students 
are taking at least one online course and 86.6% of colleges and universities now offer online 
courses (Allen & Seaman, 2013). Universities and colleges are turning to distance learning to 
meet the needs of students who seek a flexible, adaptable learning environment (Ragan, 2009). 
The surge in distance learning intensifies the need for institutions of higher education to develop 
effective evaluation processes for online faculty, as faculty evaluations can have an impact on 
teacher effectiveness, success, and growth (Berk, 2013; MacMillan, Mitchell, & Manarin, 2010; 
Wellein, Ragucci, & Lapointe, 2009). If faculty are evaluated effectively, there are a variety of 
benefits, including advancing the scholarship of teaching and learning, as well as improving the 
functionality and innovation of courses, curriculum, departments, and ultimately the broader 
community (Boyer, 1990; Glassick, Huber, & Maeroff, 1997). 

For effective evaluation to occur, evaluators must draw on multiple sources of data, 
rather than a single source, leading faculty to collaboration, reflection, and inquiry (Boyer, 1990; 
Glassick et al., 1997). Robust, well-constructed evaluations that encourage inquiry into one’s 
teaching practices can influence faculty to become rounded, productive members of the 
professoriate (Boyer, 1990; Glassick et al., 1997; MacMillan et al., 2010; Wellein et al., 2009). 
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Despite a broad acceptance that effective evaluation tools should be developed for and with 
faculty, to date, faculty evaluation systems have been largely insufficient (Arreola, 1979, 1986, 
1995, 2000a, 2000b; Arreola, Aleamoni & Theall, 2001; Berk, 2013). This is particularly true in 
the online environment where evaluation tools are often drawn from traditional settings, despite 
the arguably unique skills required to teach online (Berk, 2013; Hathom & Hathom, 2010; 
Rothman, Romeo, Brennan, & Mitchell, 2011). Research has asserted that online and traditional 
teaching techniques may be similar, but that there are key differences in evaluating online 
teacher effectiveness (Berk, 2013; Harrington & Reasons, 2005; Loveland, 2007). For instance, 
Hathom and Hathorn (2010) stated that the Web gives online instructors the opportunity to 
modify documents for students by including external links to additional resources. This is a 
unique skill to online teaching that can be evaluated by administrators. Other research has argued 
that there is little difference between teaching online and offline and therefore these evaluations 
should resemble one another (Berge & Meyers, 2000; Clark, 1989). As online education 
continues to grow and move through its generations (Moore & Kearsley, 2012), a deep and 
comprehensive understanding of online teaching and online faculty evaluation and its 
complexities is necessary. 

This paper examines a qualitative study of one university’s online full-time faculty and 
their perceptions of the tools used to evaluate their teaching. The authors provide a theoretical 
framework based in Lave and Wenger (1991) and Wenger’s (1998, 2000) notions of community 
of practice. The literature review explores faculty evaluation processes, specifically online 
faculty evaluation processes, as well as faculty commitment to evaluation processes. The study’s 
context, participants, and methodologies follow. The paper concludes with a report of the 
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findings and an analysis of the data’s broader implications for online full-time faculty and 
administrators in online departments. 

As more universities expand into online learning, it is critical to examine how faculty 
members are evaluated. Online full-time faculty’s perceptions of evaluation offer a window into 
the practices associated with evaluating faculty in this expanding learning modality. Further, it is 
necessary to gain insight into the functionality, effectiveness, and efficiency of evaluation tools 
for online full-time faculty. 

Theoretical Framework 

Communities of practice. 

Examining the evaluation processes of online full-time faculty necessitates an 
understanding of this study’s theoretical framework. Lave and Wenger (1991) and Wenger 
(1998, 2000) contend that faculty members can form a community of practice. Three components 
combine to create a community of practice, including domain, community, and practice. The 
domain is represented by a shared interest. Members of the community engage in activities and 
discussions together to pursue their domain of interest. These members of the community then 
partake in practice together, ultimately leading to shared resources and stories. Communities of 
practice form in a variety of modalities, settings, and loci. Faculty members form a community 
of practice through alignment with a social and cognitive group, as well as lived experiences. 

Learning is a key component of a community of practice. Historically, it was assumed 
that learning occurred within an individual; however, further investigation has demonstrated that 
learning is social and exists entwined with experience, activity, and community (Lave & 

Wenger, 1991; Smith, 2003). Learning consists of complex social, cultural, and historical 
systems that are accumulated and shared over time through participation within social learning 
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systems (Wenger, 2000). The survival and success of a community or organization can be 
directly related to the knowing and learning that occurs within these social systems (Wenger, 
n.d.). 

In addition to the need for organizations to thrive, there is also a desire for increased 
innovation within organizations. Because of this, groups look to improve performance. To ensure 
learning is innovative, it must occur within and amongst members of a community of practice 
(Wenger, n.d.). Furthermore, individuals in a community must engage with one another to define 
competence and to improve meaningful knowing through various elements, such as joint 
enterprise, mutuality, and shared interests (Wenger, 2000). Communities of practice have been 
described as a process where active learning takes place due to the socialization of members that 
share in similar activities, ideas, and practices (Lave & Wenger, 1991; Mayer, Grenier, Warhol, 
& Donaldson, 2013; Perrott, 2013; Ponton, 2014; Smith, 2000). In turn, learning takes place 
within a community when relationships are experienced with other members, resulting in 
ingenuity (Wenger, n.d.). 

Competence and experience can work together to generate learning and innovation 
(Wenger, 2000). In addition, organizations can encourage cross-disciplinary projects to combine 
competence and knowledge from a variety of practices to assist in the production of a project or 
goal (Hoagland, Birkenfeld, & Box, 2014; Nash, Martin, Rowell, Hetherington, & Zgliczynski, 
2011; Smith, 2003, 2009). The simultaneous learning that occurs from communities of practice 
can produce a loop that enhances learning and innovation, continuing well past the original 
project (Smith, 2003). In order for simultaneous social learning to be successful, a community of 
practice must include development that matters to individuals through a “shared repertoire of 
ideas, commitments, and memories” (Smith, 2009, para 13). Through social learning and other 
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similar interactions, individuals are bound within a relationship that cultivates trust. Because of 
this, groups are able to undertake greater and more complex activities over time (Smith, 2003). 
These processes can provide advantages to organizations like universities, as well as its faculty, 
as this brings forth greater effectiveness and success. 

Organizations like universities benefit from the social learning that can emerge from 
communities of practice (Smith, 2003). Not only do communities of practice assist in learning, 
but they also enable individuals within a group to take collective responsibility for managing 
knowledge needed to succeed. Through this understanding, organizations have found a direct 
link between learning and performance (Wenger, n.d.). When individuals learn, and do so 
collectively, performance improves. Because of this, many organizational development circles 
have shown a growing interest in utilizing community practices to overcome potential problems 
or challenges, as well as to encourage innovation and drive the application of learned knowledge 
(Hoagland et al., 2014; Nash et al., 2011; Smith, 2003). 

The goal of every university is to improve faculty to improve student learning. 
Universities can develop communities of faculty that learn from one another, engage in 
productive discussions, and collectively establish meaningful evaluation processes. This requires 
the development of processes that enhance faculty learning and investment in its programs, 
regardless of whether they teaching online or in traditional environments. Online faculty 
members are not individuals teaching in isolated environments, particularly in the environment 
discussed in this study; rather, they are a working organism, a community, a network, and a 
group that works collaboratively to ensure successful outcomes. If learning is contingent on 
community and community relies on cognitive and social experiences, then studies must 
examine how online faculty learn to be better online faculty. Universities can use online faculty 
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feedback and knowledge to develop evaluation tools that address the issues present in the 
environment. If evaluations are intended to improve online faculty performance, then it is 
incumbent upon universities to understand how online faculty members perceive the evaluation 
process and its associated documents. 

LITERATURE REVIEW 
Evaluations of Faculty 

Educational institutions use various means to evaluate the teaching performance of 
faculty members. Effectively constructed teaching evaluations allow faculty to identify strengths, 
as well as areas of opportunity in instructional practices. According to Wellein et al. (2009), an 
effective evaluation of faculty includes systematic assessment and reflective critique by various 
stakeholders, including peer, self, and specialists. MacMillan et al. (2010) contended that 
extensive evaluation mechanisms not only improve day-to-day teaching practices for individual 
instructors, they are also the first step to informed teaching and scholarship. Further, involvement 
of faculty in each step of the creation, development, and implementation has been found to be 
beneficial (Wellein et al., 2009). 

Effective evaluation provides clarity on faculty members and their progress towards 
specific goals and targets. A range of evaluations is needed to gather a comprehensive view of 
faculty instruction (Wellein et al., 2009). According to Wellein et al. (2009), self, specialist, 
student, and peer evaluation formats can be effective. Different formats provide different 
perspectives, thereby offering a holistic view of faculty. Each type of evaluation has unique 
objectives. The purpose of specialist review is to evaluate a specific skill or skill set (Wellein et 
al., 2009). Through expert supervision, faculty may receive feedback that allows for exploration 
of new teaching material and teaching requirements. Peer evaluations have been found to 
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increase faculty implementation of active learning and critical thinking skills within lectures and 
lessons (Wellein et al., 2009). The reflective model used during self-evaluation allows faculty 
members to participate in critical inquiry. Finally, student evaluations have been found to be one 
of the most commonly utilized methods to assess faculty performance (Wellein et al., 2009). 
Student evaluations can provide information regarding teacher effectiveness as well as teacher 
preparedness (Wellein et al., 2009). Effective evaluation processes should be designed to affect 
teaching and resulting learning and to communicate this evidence in a public forum. 

Evaluations of Online Faculty 

While there is an extensive body of research related to the evaluation of faculty in 
traditional settings, there have been fewer studies examining online faculty members’ self- 
reported perceptions of evaluation processes. Due to the rapid growth of online education, the 
existing evaluation scales, such as those used in traditional instructional settings, have been 
questioned (Berk, 2013; Eskey & Schulte, 2012; Hathom & Hathorn, 2010; Mandernach, 
Donnelli, Dailey, & Schulte, 2005; Rothman et al., 2011; Schulte, 2009; Tobin, 2004). As these 
evaluation tools were challenged, concern grew that their relevance and efficiency within the 
online classroom may not be accurate, effective, or sufficient (Berk, 2013). Creasman (2012) 
identified a number of differences in instruction in the online environment. Such differences 
include the asynchronous style of environment, the non-linear forums that allow students to 
participate in several discussions at one time, student-teacher communication, and an increased 
volume of information. Because of this, the complexities of the online environment must be 
considered when creating and implementing evaluations for online educators (Berk, 2013; 
Harrington & Reasons, 2005; Loveland, 2007). 
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Universities around the United States have moved to the online full-time faculty model to 
increase student retention and long-term success along with faculty satisfaction, including 
Southern New Hampshire University, Grand Canyon University, University of Maryland 
University College, and American Public University System, amongst others (Fain, 2011; SNHU 
Communications, 2013). Several institutions claim a unique online full-time faculty model. 

Some models require that faculty meet at a particular facility to hold office hours and engage in 
training and professional development while others require online office hours and online 
training opportunities (Fain, 2011). Concerns about adjunct faculty pay and job security, as well 
as a desire for student retention, have helped lead to the development of online full-time faculty 
models (Mueller, Sanderson, & Mandernach, 2013); however, there is little research on the 
evaluation of the faculty (Author et al, 2014). While the models are still in their infancy at post¬ 
secondary institutions, as online education develops, it becomes increasingly important to 
understand how to best evaluate online full-time faculty. What constitutes an “effective 
evaluation process” for online faculty is an area that continues to require additional study. 
Faculty Engagement in the Evaluation Process 

Although there is a need for more literature on the evaluation processes of online faculty, 
Baran, Correia, and Thompson (2011) claimed that institutions of higher education should 
consider “teachers as adult learners who continuously transfonn their meaning of structures 
related to online teaching through a continuous process of critical reflection and action” (p. 421). 
This suggests that an effective evaluation of faculty need not only lie in the hands of 
administrators. Indeed, “The process of teacher evaluation in institutions of higher education 
should be an organic whole” (“A preliminary look,” 2005, p. 49). Critical reflections from 
faculty are integral to the development of a comprehensive evaluation program. 
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Successful performance evaluation is found in evaluations that are both acceptable to the 
rater and ratee (Szeto, 1994). Furthermore, for faculty members, and ultimately their students, to 
be invested and engaged in their community of practice, they must understand how and why they 
are evaluated. They must feel a degree of commitment to the evaluation process and to 
improving teacher performance. Research has shown that using a bottom-up process will allow 
faculty to obtain a sense of ownership in the standards and evaluation process; thereby, creating 
substantial legitimacy amongst faculty members (Galluscio, 1998). This serves in contrast to a 
top-down model, which relies heavily on administrator evaluation. Commitment to a common 
community and an increased acceptance of evaluation can be enhanced by involving faculty in 
the development of evaluation procedures (Szeto, 1994). Further research is required to 
determine faculty’s perceptions of the characteristics of an effective online instructor, ideal 
evaluation methodologies, as well as concerns about their evaluation. 

METHODOLOGY 

Context 

The online full-time faculty model at the university where the study took place is atypical 
of most institutions with online programs. While many colleges and universities have begun to 
develop online full-time faculty models, the researchers believe location, work requirements, and 
faculty oversight make this model unique; this is due to the researchers’ prior experiences at 
other institutions and their current experiences at the institution under study. The model includes 
undergraduate and graduate instructors teaching online in a rolling enrollment program. Each 
instructor teaches approximately four courses at a time and has four computer monitors to view 
documents, assess student work, note phone calls, and engage with students in the discussion 
forums. Notwithstanding their status as online faculty, instructors hold office hours eight hours a 
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day Monday through Friday in a building with other online full-time faculty members, as well as 
students’ counselors and support staff. The faculty members are responsible for responding to 
student calls, emails, and messages, as well as grading assignments, guiding classroom 
instruction, and preparing curricular materials for dissemination in the classroom during office 
hours. They are also expected to communicate with traditional faculty and students’ counselors. 
Instructors are encouraged to participate in professional development opportunities, as well as 
scholarly activities, including research and publication. Each faculty member reports to a 
supervisor and director who conduct informal weekly and quarterly reviews, as well as a formal 
annual review. The supervisor supports a team of online full-time faculty while also teaching a 
section or more of a course. The faculty team is often content focused; however, there are 
exceptions to this due to small numbers of faculty in particular content areas and the supervisor’s 
area of content expertise. The supervisor is responsible for supporting the curriculum 
improvement process in addition to evaluating faculty, offering student support, and initiating a 
range of program improvements within the content area and within the department and larger 
college. 

The program has relied substantially on supervisor evaluation of faculty with an analysis 
of at least one course taught per quarter. The quarterly review process was conducted by 
supervisors and served as a convention to formatively assessment and improve practice. The 
document under review in this study included a list of 25 criteria related to the areas of 
participation, engagement, and facilitation; grading and feedback, classroom management; and 
personal development and relationships. Faculty members were rated by supervisors as “met,” 
“partially met,” or “did not meet” for all 25 criteria. The supervisor was expected to offer 
documentation along with the ranking. A summative evaluation of performance was offered at 
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the conclusion of the document where the online full-time faculty member was ranked as 
“exceptional,” “good,” or “needs improvement.” 

Because communities of practice suggest the need for thorough, rounded evaluations to 
inform teacher practice and growth, the program described in this study underwent a formal 
program evaluation to explore the evaluation processes of online full-time faculty members. The 
research team established a single objective: to collect feedback from online full-time faculty 
members regarding how they are evaluated and to use this data to improve the university’s online 
full-time faculty quarterly evaluation processes. The research question for the present study 
asked, “To what degree does the university’s online full-time faculty’s evaluation processes align 
with what online instructors perceive as useful and supportive of their efforts as teachers?” 
Participants 

The research team involved stakeholders directly invested in the development of faculty 
and the impact of teaching on student learning in the online environment. There were six 
members on the research team, including directors, supervisors, and faculty. Three of the 
researchers were part of the administrative team who directly evaluated faculty. The researchers 
were clear in both purpose and expectation of this study, namely that the process was for 
research purposes and improvement initiatives for the online full-time faculty department. 
Researchers exercised transparency by informing participants that the survey was anonymous 
and no one would be able to establish the identities of those involved. 

In the first quarter of 2014, all 169 online full-time faculty members at a large university 
in the Southwest were invited via email to participate in a survey. One hundred and eighteen of 
the 169 faculty participated in the survey. This is a response rate of 69.8%. The response rate 


55 


The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 

31 

may have been influenced by the small-scale pilot study administered prior to the larger survey 
sent to faculty. 

Of the 118 faculty who responded to the survey, 41.53% had been teaching at the 
university level for 2-5 years, 44.07% had been an online full-time faculty member at this 
university for 2-5 years (zero had been in this position at the university for more than five years 
because the position was not created until 2010). The study participants included faculty from 
the doctoral, education, arts and sciences, theology, and business colleges teaching 
undergraduate, masters, and doctoral level courses. The research team members opted not to 
collect additional demographic information on categories like gender, age, racial and ethnic 
identity, or religion because of the researchers’ intimate knowledge of the faculty, ultimately 
ensuring participant anonymity and reducing the potential for researcher bias. 

Method 

The study described in this paper was qualitative in design. A small-scale pilot survey 
was disseminated via email to a random stratified group of 44 online full-time faculty members 
from each college at the university. Survey Monkey, a web-based survey service, was used to 
administer the instrument. The survey was primarily qualitative in nature, asking open-ended 
questions and was distributed prior to the large-scale study to identify gaps in the survey 
instrument. Results from the pilot study resulted in two changes: the research team clarified the 
wording on one of the questions and included the current document used by supervisors to 
evaluate faculty for reference. 

A revised follow-up survey was sent via email and Survey Monkey to all 169 online full¬ 
time faculty members at the university where the study took place. This study was Institutional 
Review Board approved. Faculty members were told that their participation was voluntary and 
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anonymous. Further, faculty members were not required to answer every question on the survey. 
Participants completed the survey in approximately 20 minutes and were given two weeks to 
complete the survey until the link was closed. Both the pilot survey and follow-up survey were 
approved by the university’s Institutional Review Board. 

The survey asked descriptive questions regarding online teaching and the evaluation 
processes of online instructors. The instrument was divided into three sections, including: (1) 
perceptions of the roles of online faculty, (2) perceptions of teaching evaluations, and (3) 
perceptions of the current evaluation processes for online full-time faculty. See the list of 11 
descriptive survey questions in Appendix A. The first section, perceptions of the roles of online 
faculty, contained items related to qualities of an effective online instructor, identity as an online 
instructor, and areas of growth or opportunity as an online instructor. The second section, 
perceptions of teaching evaluations, included items related to the importance of evaluating online 
instructors, most beneficial types of evaluations, and an ideal evaluation of teaching. The third 
section, perceptions of the current evaluation processes for online full-time faculty, contained 
items regarding the current evaluation processes for faculty at this university, elements to revise 
in the current process, and the effectiveness of the current process. There were 11 descriptive 
questions in total. 

The descriptive survey questions were analyzed qualitatively through the content analysis 
method (Carney, 1972; Holsti, 1968, 1969; Krippendorff & Bock, 2008). The researchers were 
not seeking to analyze quantitative measures like means, standard deviations, or significance. 
Rather, analysis involved identification of robust codes, which describes codes that are most 
prominent in a textual data set. Qualitative content analysis is a specific method of analysis that 
allows for the counting of codes to draw conclusions and extrapolate findings (Krippendorff & 
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Bock, 2008). Codes were counted to simply classify the themes and identify occurrences of units 
of text, but the emphasis was on the textual concepts and trends that emerged from the findings. 
The hope was that the emergent themes would illuminate trends and phenomena taking place 
within the department, which would ultimately inform decisions made regarding the evaluation 
processes of online full-time faculty. 

The analysis process was systematic and purposeful. The team reviewed 11 descriptive 
survey responses from 118 full-time online faculty members, highlighting and notating each unit 
of analysis relevant to the research question. Units of analysis included descriptive words, 
phrases, and sentences. After the initial analysis, similar units were combined. These units were 
then collapsed into other larger categories based on similar content or redundancies. After this 
step, key words or phrases from the units were extracted, resulting in a set of code or categories 
for each descriptive question. The process continued until all relevant units were grouped or 
regrouped with similar units and labeled with a code (Krippendorff & Bock, 2008). The team 
then identified robust themes by counting instances for frequency (Krippendorff & Bock, 2008). 
For the purposes of this analysis, any code with more than seven units was considered robust. 
There was a decline after seven units for each code; thus, the selection of seven was intentional, 
not arbitrary. Codes with less than seven units were considered weak and not included in the 
findings and analysis. 

The team analyzed each set of survey responses independently to develop codes with as 
little bias as possible, focusing on the words scribed by survey respondents. The six researchers 
each shared his or her codes through a coding workshop designed to ensure intercoder reliability 
(Miles & Huberman, 1994; Neuendorf, 2002), a term used frequently in qualitative research to 
ensure independent coders agreed on content coding and structures. The workshop afforded 
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researchers the opportunity to identify points of conflict or communion in the coding process, to 
move codes into new categories, to alter the language of categories if needed, and to agree upon 
robust codes. For example, one of the researchers had identified a code in her individual coding 
session; however, after group discussions, it was determined that the label was not specific 
enough. As such, the group developed a new label in concert to describe the phenomenon. A 
series of robust codes emerged from the coding workshop and are explicated in the findings and 
analysis section. 

Findings and Analysis 

For the purposes of brevity and clarity, robust codes from three of the 11 descriptive 
questions are detailed. For more detailed findings from one specific survey question, please 
reference Author 1 et al. (2014). These three questions were selected for discussion because they 
most align with the study’s primary research question: “To what degree does one university’s 
online full-time faculty’s evaluation processes align with what online instructors perceive as 
useful and supportive of their efforts as teachers?” The three survey questions analyzed included: 

(1) “Do you feel the current supervisor evaluation document is useful to you? Why or why not?”; 

(2) “Do you feel the current supervisor evaluation process supports your work as a teacher? Why 
or why not?”; and (3) “If you could envision the ideal process to evaluate your teaching, what 
might that process look like? How frequently would you be evaluated?” 

For the survey question, “Do you feel the current supervisor evaluation document is 
useful to you? Why or why not?,” 57 participants responded with positively aligned comments, 
40 with negatively aligned comments, 14 stated there were positive and negative elements to the 
document, and eight stated they had never seen the document. No robust codes emerged from 
responses with both positive and negative comments regarding the document. See Table 1. 
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The most robust code for positively aligned responses was that the document identifies 
areas to improve and reflect. There were 23 units in this code. When describing whether the 
document was useful, faculty expressed comments such as, “Gives me areas to improve,” “sets 
goals and reflects on my progress,” “it points out areas of improvement,” “reminds me of 
strengths and weaknesses,” “helps find area of improvement,” and “indicates where 
improvement can be made.” Faculty members’ comments suggested that the majority of online 
full-time faculty at this university believed the current supervisor evaluation document offered 
opportunities to reflect on specific areas of their teaching that needed improvement. 

The second most robust code for positively aligned comments was that the document 
established clear expectations for their performance. There were 15 units in this code. When 
describing whether the document was useful, faculty expressed comments such as, “Useful to 
meeting all my expectations,” “allows me to refer back to my goals and expectations regularly,” 
“useful reflection on what is required by the university,” and “good to know how the university 
feels that one is doing.” Faculty members’ comments established that the document provided 
clarity into expectations as it relates to their job performance and the requirements of the 
university’s online full-time faculty department. 

The most robust code for negatively aligned responses was that the document focused on 
operational tasks, not teaching quality. There were 12 units in this code. When describing why 
the document was not useful, faculty expressed comments such as, “It just measures the 
performance of operational tasks,” “it is just a fixation on details,” “does not focus on what is 
really important,” “tremendous amount of emphasis on what you have or have not completed in 
relation to your requirements,” “all it measures is quantifiable data,” “spends too much time on 
numerics,” “I don’t like the more robotic expectations lists,” and “I already know whether or not 
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I have completed administrative tasks.” Faculty members’ comments suggest that some found 
the document helpful for clarifying expectations while others found the document to be heavily 
focused on quantitative, detailed, or “robotic” measurements, as opposed to a holistic portrait of 
the classroom. Further, the data demonstrated that the evaluation document did not emphasize 
teaching quality and “what is really important,” although those faculty did not specific what 
qualified as “important” in the online classroom. 

The second most robust code for negatively aligned responses was that the document is 
top-down or operationally oriented. There were nine units in this code. When describing why the 
document was not useful, faculty expressed comments such as, “Possible manipulation of the 
document to be for or against an employee,” “they are set up for failure,” “instructors feel over 
watched,” and “only tells me what my supervisor saw; no other information is new news.” 
Faculty members’ comments suggested that these nine faculty not only found the document 
misguided and focused on administrators, but also reported an approach that was misaligned with 
the freedoms they associate with membership in the academy, as noted by comments related to 
being “watched” by supervisors. 

Table 1 


Survey Question 

Robust Code 

Number of 
Units 

Do you feel the current supervisor 
evaluation document is useful to 
you? Why or why not? 

Identifies areas to improve and reflect 

23 

Established clear expectations for their 
performance 

15 

Focused on operational tasks, not 
teaching quality 

12 

Top-down or operationally oriented 

9 


For the following question, “Do you feel the current supervisor evaluation process 


supports your work as a teacher? Why or why not?,” 41 answered with positively aligned 
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comments, 34 with negatively aligned comments, and 23 with comments stating both positive 
and negative elements of the process. See Table 2. 

The most robust code with positively aligned responses was identifies areas for 
improvement or opportunity in instruction. There were 21 units in this code. When describing 
why the current process supported their teaching, faculty expressed comments such as, “Helps 
improve my teaching and practices,” “directions on areas that I can become better in,” “makes 
me a better teacher,” “challenges me to be better,” “lets me kn ow what I’m doing well,” “shows 
me what I can improve on,” “provides ideas for instructional improvement,” and “gives 
instructional feedback.” Faculty members’ comments suggested that the current process 
uncovered specific areas or “directions” an instructor can move to improve their instructional 
techniques and generally “be better.” The focus on being challenged to move forward in their 
online teaching was strong. The faculty stressed how the process helped them as individual 
instructors improve, perhaps ultimately influencing student learning. 

The second most robust code with positively aligned responses was identifies degree to 
which expectations are met. There were seven units in this code. When describing why the 
current process supported their teaching, faculty members expressed comments such as, “Ensure 
I’ve met necessary components,” “meeting my expectations,” “meeting my grading 
expectations,” and “reviews time spent in cloud, met requirement.” Similar to the first research 
question regarding the effectiveness of the evaluation document, faculty noted that the process 
illuminated whether the expectations of their work as online full-time faculty members were met 
and the degree to which they were met. 

The most robust code with negatively aligned responses was focuses on expectations 
rather than instructional practices. There were 13 units in this code. When describing why the 
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current process did not support their teaching, faculty expressed comments such as, “A faculty 
checklist,” “a task scorecard,” “review of the standard expectations,” “checklisty,” “no feedback 
on actual teaching practices,” “no focus on pedagogical content knowledge of instructor,” and 
“not actually evaluating the teaching or potential teaching of an instructor.” These faculty 
members expressed frustration or anxiety related to the lack of emphasis placed on content 
knowledge and the strong emphasis on the “checklist” structure of the evaluation document. 

The other equally robust code with negatively aligned responses was the role of 
management and policies in process. There were 13 units in this code. When describing why the 
current process did not support their teaching, faculty expressed comments such as, 

“Management documentation of my performance,” “I never see the results,” “those that review 
me do not teach my course,” “not connected to learning experience, only to administration,” and 
“a piece of paper is just a piece of paper.” The faculty members who were concerned with the 
process noted concerns with the procedures associated with the dissemination of the document, 
as well as the experience of the supervisor completing the evaluation. 

Several responses offered both negatively and positively aligned responses, creating one 
robust code. The robust code in this category was that evaluation processes need to be revised. 
There were 18 units in this code. When describing why the current process was somewhat 
supportive of their teaching, faculty members expressed comments such as, “One should be able 
to achieve excellence in reviews,” “better served with specific reviews based on subject being 
taught,” “too micro-focused,” “helpful if done in a way to help us grow as educators,” “should 
include ways to measure new and innovative teaching strategies,” and “training would be more 
effective.” These faculty members reported an appreciation for an element of the process but 
explained that it was inadequate because there were opportunities for revision in the following: 
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an improved scoring system, more content-focused comments from evaluators, and enhanced 
procedures used to disseminate the evaluation. One individual offered a different approach, 
specifically a faculty training system, rather than supervisor evaluation. 

Table 2 


Survey Question 

Robust Code 

Number of 
Units 

Do you feel the current supervisor 
evaluation process supports your 
work as a teacher? Why or why not? 

Identifies areas for improvement or 
opportunity in instruction 

21 

Evaluation processes need to be 
revised 

18 

Focuses on expectations rather than 
instructional practices 

13 

The role of management and policies 
in process 

13 

Identifies degree to which 
expectations are met 

7 


For the following question, “If you could envision the ideal process to evaluate your 
teaching, what might that process look like? How frequently would you be evaluated?”, the most 
robust code was that evaluations should/bmv on growth of the instructor and students. There 
were 33 units in this code. When describing the ideal process, faculty expressed comments such 
as, “Desire for growth,” “less task-y or checklist-y,” “evaluate the effectiveness of the instructor, 
not the mundane faculty activities,” “on level of learning that takes place,” “more qualitative and 
personal,” “given specifics on how to improve,” “challenge critical thinking and deeper 
thinking,” “evaluation of use of higher order thinking,” “focus on growth of employee,” 
“promote ongoing growth,” “qualitative rather than quantitative,” and “show areas of growth.” 
Faculty reported that the ideal process should be qualitative, personal, less focused on a 
checklist, and intended to grow the faculty’s online teaching techniques and ensure student 
learning. Those who envisioned a new system expressed their “ideal” process in contrast to the 
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system currently in place to evaluate their teaching and expressed a desire for qualitative, 
holistic, and inquiry-based feedback. 

The second most robust code was that administrators should select evaluators that can 
effectively evaluate courses. There were 14 units in this code. When describing the ideal process, 
faculty expressed comments such as, “Supervisors may not have the training or experience in my 
specific field to provide adequate assessment,” “faculty to meet with one another to share best 
practices,” “360 style,” “evaluated by a subject matter expert,” “evaluators who know the content 
to evaluate a class,” “faculty meet with one another to share best practices,” and “someone who 
is capable of instructing my content should evaluate me.” These faculty members noted interest 
in being reviewed by a peer or supervisor with subject matter expertise and the ability to share 
best practices within a particular content. 

Of those who noted frequency in their responses, 26 or 41% preferred bi-annual 
evaluations. Twenty or 32% preferred annual supervisor evaluations. Seventeen or 27% 
preferred quarterly supervisor evaluations. 

Table 3 


Survey Question 

Robust Code 

Number of 
Units 

If you could envision the ideal 
process to evaluate your teaching, 
what might that process look like? 
How frequently would you be 
evaluated?” 

Focus on growth of the instructor and 
students 

33 

Select evaluators that can effectively 
evaluate course 

14 


DISCUSSION 

Findings from this study have several implications. Above all, online full-time faculty 
members want to grow as instructors. Although this is not surprising data, it has four functions. 
First, it provides an initial baseline for understanding how online full-time faculty members 
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perceive evaluation processes. Online full-time faculty departments are a new but growing 
phenomenon in higher education (Fein, 2011; SNHU Communications, 2013), specifically the 
model used at the university described in this study. Because this department is unique in regards 
to proximity and oversight, the baseline serves as the foundation for universities that resemble 
the one discussed in this study who are developing evaluation processes for online faculty. 
Further, this study provides insight into program evaluation processes that universities can 
undergo to potentially improve faculty satisfaction or, at minimum, involve faculty more in 
decision-making. 

Second, on li ne full-time faculty members in this study rarely expressly focused on 
modality. “Growth” was stated in general terms regarding a desire to be a better pedagogue for 
student learning, not necessarily a desire to be a better instructor in the online modality. This 
does not mean that the faculty conceived of their roles or evaluation as online instructors as 
unimportant. This does mean, however, that this population preferred evaluations focused on 
content and teaching practices. The study’s contributes the notion that while differences in online 
and traditional teaching may be clear to some stakeholders, it may not be tangible or significant 
to online full-time faculty. In this setting, modality did not appear to be a factor in faculty’s 
perceptions of evaluation. Faculty members strongly emphasized general instructional 
improvements in their responses. Online full-time faculty’s perceptions, specifically the robust 
codes, are arguably applicable to all instructors, regardless of modality. Stakeholders can use this 
knowledge when developing criteria for online faculty evaluations. More research is needed to 
understand what it means to encourage growth in online instructors and what role modality plays 
in instructors’ concepts of online teaching, particularly in settings like the one described in this 
study. 
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Third, the emphasis on growth suggests that online full-time faculty members are eager to 
refine their skills as instructors and want the evaluation tool to mirror their desire for growth, not 
only capture the degree to which they have or have not met job expectations. Respondents 
emphasized a desire to be challenged and to improve their teaching skill set, while 
deemphasizing the role of checklists, tasks, and expectations. Supervisors in computer-mediated 
teaching departments can use this information when disseminating evaluation results and when 
developing plans to lead their teams towards instructional improvements. 

Finally, while faculty members in this study identified positive elements of the process, 
some felt that the current evaluation document was not sufficient, the current evaluation process 
did not fully support their work as faculty, and the ideal process for evaluating online full-time 
faculty did not align with the process currently in place. The current evaluation document was 
developed by a supervisor and director team and did not include faculty insight. The document 
included three findings for each criteria, “met,” “partially met,” or “did not meet,” and 25 criteria 
related to (1) pre-course set up requirements, (2) participation, engagement, and facilitation, (3) 
grading and feedback, (4) classroom management, and (5) personal development and 
relationships. Some of the criteria included holistic measures, while several included 
quantitative-focused measures such as response times to student queries and posts made in 
particular forums. Upon hire and throughout an online full-time faculty’s tenure, he or she 
partakes in one-on-one coaching from a faculty trainer on evaluation expectations, as well as 
continued education through online training modules and face-to-face training related to 
evaluation expectations. 

Faculty generally expressed that they were aware of evaluation expectations; however, 
they noted concerns that the current document and evaluation process weighed heavily on 
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expectations or quantitative data, as well as noted the limitations of the top-down approach of 
supervisor evaluation and evaluation by individuals without specific content knowledge. This 
finding has implications for administrators in computer-mediated teaching departments when 
developing evaluations, ideally focused on qualitative or holistic measures disseminated through 
a rounded approach by content experts. 

Although the negatively aligned comments may appear to be simply critiques of the 
department’s evaluation processes, the research team sees otherwise. The department was 
established in the latter half of 2010, and it is an organism that seeks improvement in all 
programmatic areas from curriculum to assessment of students to faculty working conditions to 
evaluation processes. This survey data confirms the need to continue to improve the 
department’s evaluation document and process with an emphasis on the growth of faculty. 
Because of the department’s distinct structure, there is a built-in community of practice, one that 
focuses on reflection and innovation (Lave & Wenger, 1991; Wenger, 2000). It is incumbent 
upon the department and other computer-mediated teaching departments to ensure that 
evaluation documents and processes align with the community and to understand how physical 
proximity might influence perceptions of online teaching evaluations. 

These findings led the research team to reflect upon next steps, ultimately choosing to 
collect additional data from faculty. The research team deemed it necessary, based on the 
outcome of the present study, to establish a focus group and workshop to further understand the 
strengths and weaknesses of the current evaluation system. This process was designed to 
integrate researchers, faculty, and administrators, working collaboratively to revise the 
evaluation document and process. These next steps are multi-fold and rooted in the framework 
and findings of this study. The focus group and workshop were intended to enhance faculty’s 
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community of practice in order to lead to enhanced knowledge of online teaching, greater 
collaboration, and improved student learning (Garrison, Anderson, & Archer, 2000, 2010; Lave 
& Wenger, 1991; Wenger, 1998, 2000). Additionally, as the data demonstrated, a significant 
number of faculty members expressed concern about the emphasis placed on quantitative 
measures, as well as the individual completing the evaluation. The research team identified the 
opportunity for growth in the evaluation processes of online full-time faculty and chose to gather 
additional faculty input on these concerns. This new study aligns with the literature suggesting 
that faculty evaluations should be comprehensive and involve faculty insight, as well as the 
literature stating that faculty inquiry and reflection can lead to greatest growth in student learning 
(MacMillan et al., 2010; Wellein et al., 2009). 

Because of the study described in this manuscript, as well as the focus group and 
workshop study, the online full-time faculty department intends to revise the evaluation 
instrument and process to align with faculty’s perceptions and to meet their needs as scholars of 
teaching and learning (Boyer, 1990; Glassick et al., 1997). The newly developed evaluation 
instrument will seek to include useful components from the original document, while 
incorporating components focused on teacher effectiveness, growth, and the opportunity for 
evaluators to provide qualitative, holistic feedback. In addition, the new evaluation document 
and process will use peer, supervisor, and self-evaluations rather than focus solely on supervisor 
evaluation. The research team intends to collect quantitative and qualitative data on the 
effectiveness of this new process. As the new document and process are developed, 
implemented, and refined, additional research on online full-time faculty’s perceptions of 


evaluations will follow. 
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Appendix A 

Online Full-Time Faculty Descriptive Survey Questions 

Perceptions of the Roles of Online Faculty 

1. What do you believe makes a quality online instructor? 

2. Do you believe you are a quality online instructor? Why or why not? 

3. What areas of growth are there (if any) in your development as an online instructor? 

Perceptions of Teaching Evaluations 

4. If evaluations of online instructor must exist, what ki nds or types of evaluations would be 
most beneficial to you? 

5. If you could envision the ideal process to evaluate your teaching, what might that process 
look like? How frequently would you be evaluated? 

Perceptions of the Current Evaluation Processes for Online Full-Time Faculty 

6. What do you think of the document’s name “Faculty Support Review”? To what degree 
is this an accurate description of the process? 

7. Would you change the name of the Faculty Support Review document or keep the name 
the same? If you would change the document’s name, what names do you suggest? 
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8. In reviewing your past Faculty Support Reviews, do you believe there are elements 
missing or that need to be removed from the document? If so, what are they and why? 

9. Do you feel the Faculty Support Review is useful to you? Why or why not? 

10. Do you feel the Faculty Support Review process supports your work as a teacher? Why 
or why not? 

11. Are there any other comments, questions, or feedback you would like to share? 

Note: Language in the descriptive survey questions above was adjusted to protect the anonymity 
of the institution and department. 


